Ar-based supplementary teaching system for guzheng and method thereof

ABSTRACT

An AR-based supplementary teaching system for guzheng and method thereof, the system includes an AR device, a data processing device and positioning devices for key positions, the data processing device is signal-connected to the AR device, and the positioning devices is installed on the guzheng code of guzheng, the positioning devices corresponds to the guzheng code of guzheng one by one; the AR device is used to obtain real scene data; the data processing device is used to guzheng and the positioning devices identify and generate string distribution data; also used to obtain operation instruction based on user actions, execute the operation instruction and generate virtual data; the AR device is also used to convert all data based on the string distribution data The virtual data and the real scene data are superimposed and displayed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Non-provisional application claims priority under 35 U.S.C. §119(a) on Chinese Patent Application No(s). 2019112391282 filed on Dec.6, 2019, the entire contents of which are hereby incorporated byreference.

BACKGROUND OF THE INVENTION Technical Field

The invention relates to a field of virtual reality, in particular to anAR-based supplementary teaching system for guzheng and method thereof.

Description of the Related Art

At present, the guzheng teaching human-computer interaction auxiliarydevice and system (CN109637271A) discloses an auxiliary device that canprovide basic fingering instruction for guzheng, as well as practicefeedback guidance, to avoid problems such as confusion, irregularfingering, and uncoordinated left and right hands. But the immersion ispoor and requires a lot of peripheral devices to achieve, especially thelight bar indicating device, and the light bar indicating device islocated below the guzheng code, so replacing the light bar indicatingdevice requires re-tuning the guzheng, which is poor in portability andcumbersome to use, the user experience is low.

BRIEF SUMMARY OF THE INVENTION

In view of the shortcomings in the prior art, the present inventionprovides an AR-based supplementary teaching system for guzheng andmethod thereof that can improve user immersion.

In order to solve the above technical problems, the present invention issolved by the following technical solutions:

An AR-based supplementary teaching system for guzheng, including an ARdevice, a data processing device, and a plurality of positioning devicesfor key positions, wherein the data processing device is signallyconnected to the AR device, the positioning devices for key positionsare installed on a guzheng code, and each of the positioning devices forkey positions corresponds one-to-one with the guzheng code;the AR device is used to obtain a real scene data;the data processing device is used to identify the guzheng and thepositioning devices for key positions in the real scene data, generatinga string distribution data; the data processing device is also used toobtain an operation instruction based on user's actions, executing theoperation instruction, and generating a virtual data; andthe AR device is also used to superimpose and display the virtual datawith the real scene data based on the string distribution data.

In an implementable embodiment, the system further includes a soundinput device, the sound input device is signally connected to the dataprocessing device;

the voice input device is used to collect a sound data when the userplays and send the sound data to the data processing device;the data processing device is configured to identify the sound data,generate a tone data, retrieve the operation instruction associated withthe tone data, execute the operation instruction, and generate thevirtual data.

In an implementable embodiment, the data processing device includes atransmission module, an identification module, a storage module, and aprocessing module, wherein the identification module is signallyconnected to the sound input device, the transmission module, thestorage module, and the processing module respectively, the processingmodule is also signally connected to the transmission module and thestorage module respectively, and the transmission module is signallyconnected to the AR device;

the identification module is used for receiving and identifying thesound data to obtain the tone data, and for receiving and identifyingthe real scene data to obtain the string distribution data;the processing module is used for retrieving the operation instructionassociated with the tone data from the storage module, executing theoperation instruction, and generating the virtual data according to theoperation instruction; and also, for controlling the AR device tosuperimpose and display the virtual data with the real scene data;the virtual data includes a string guidance data, a fingering guidancedata, and a numbered musical notation display data.

In an implementable embodiment, the system further includes asomatosensory device, the somatosensory device is signally connected tothe data processing device;

the data processing device also used to generate a fingeringsomatosensory guidance data according to the operation instruction, andcontrol the somatosensory device to perform a somatosensory feedbackaccording to the somatosensory guidance data.

In an implementable embodiment,

the somatosensory device includes a wristband and four smart fingers;each of the smart fingers is provided with a pick installation sleeve, afirst vibration feedback device and a second vibration feedback device,wherein all first vibration feedback devices are located on a same sideof the pick installation sleeves, all second vibration feedback devicesare located on an opposite side of the pick installation sleeves, andeach of the pick installation sleeves is used to install a pick;the wristband is provided with a somatosensory transmission module, abattery module, and a control module, wherein the battery modulesupplies power to the first vibration feedback devices, the secondvibration feedback devices, the somatosensory transmission module, andthe control module; the control module is signally connected to thefirst vibration feedback devices, the second vibration feedback devices,and the somatosensory transmission module respectively; and thesomatosensory transmission module is signally connected to thetransmission module;the control module is used for receiving fingering somatosensoryguidance data by the somatosensory transmission module, and controllingthe first vibration feedback devices and the second vibration feedbackdevices to perform a vibration feedback according to the fingeringsomatosensory guidance data.

In an implementable embodiment,

wherein a lower surface of each of the pick installation sleeves isprovided with an upper mounting member, a first pressure sensor, and asecond pressure sensor; the first pressure sensor is located on a sideaway from the pick, and the second pressure sensor is located on a sideclose to the pick side, the first pressure sensor and the secondpressure sensor are both signally connected to the control module;wherein an upper surface of each of the smart fingers is provided with alower mounting member, a first trigger, and a second trigger; the firsttrigger corresponds to the first pressure sensor, and the second triggercorresponds to the second pressure sensor; the lower mounting member andthe upper mounting member are rotatably connected, so that when the pickmove the string, the first pressure sensor squeezes the first trigger orthe second pressure sensor squeezes the second trigger; wherein thecontrol module is used to receive a pressure data detected by the firstpressure sensor and/or the second pressure sensor, and send the obtainedpressure data to the data processing device by the somatosensorytransmission module, and the data processing device analysis a fingeringaccuracy according to the pressure data.

In an implementable embodiment, each of the positioning devices for keypositions includes an identification panel, a connecting member, and afixing member; the fixing member is provided with a notch matching ashape of the guzheng code, the fixing member is installed on an upperend of the guzheng code through the notch during positioning; and

the identification panel is connected to the fixing member through theconnecting member.

In an implementable embodiment, the system further includes a cloudserver;

the cloud server is in signal connection with a plurality of dataprocessing devices, and each of the plurality of data processing devicesperforms data interaction with other data processing devices through thecloud server.

In order to solve the above technical problems, the present invention isalso provided with an AR-based supplementary teaching method is realizedby using any of the above-mentioned system, wherein the method includesthe following steps:

setting a positioning device for key positions on each guzheng code ofthe guzheng;obtaining the real scene data through the AR device, identifying theguzheng and the positioning devices for key positions in the real scenedata, and generating the string distribution data;displaying an operation interface for the user by the AR device,obtaining the operation instruction according to the user's actionssimultaneously, executing the operation instruction and generating thevirtual data, sending the virtual data to the AR device, and the ARdevice superimposing and displaying the virtual data with the real scenedata based on the string distribution data.

In an implementable embodiment, the step of displaying an operationinterface for the user by the AR device, obtaining the operationinstruction according to the user's actions simultaneously includes:

displaying an operation interface for the user by the AR device, andsimultaneously collecting and identify a sound data to obtain a tonedata; obtaining an operation data corresponding to the tone dataaccording to a pre-defined operation association rule, and generatingoperation instruction based on the operation data;the operation instruction includes a playing guidance instruction and anevaluation instruction; when the operation instruction is the playingguidance instruction, the virtual data includes a string guidance data,a fingering guidance data, and/or a numbered musical notation displaydata, at which time the operation instruction is executed to generatethe virtual data include:acquiring the operation data according to the user's actions, andextracting a corresponding music score data based on the operation data,the music score data includes a note data and a fingering data; andgenerating a string guidance data, a fingering guidance data and/or anumbered musical notation display data according to the playing guidanceinstruction, including:

-   -   extracting a corresponding string position according to the note        data based on the pre-defined string association rule, and        generating a string guidance data;    -   extracting a corresponding fingering 3D animation based on the        pre-defined fingering association rule according to the        fingering data, and generating a fingering display guidance        data; and    -   extracting a musical notation display data corresponding to the        music score data.

In an implementable embodiment, when the operation instruction is theplaying guidance instruction, a somatosensory guidance step is alsoincluded, and the somatosensory guidance step include:

based on the pre-defined fingering association rule, the correspondingsomatosensory feedback data is extracted according to the fingering datato generate a fingering somatosensory guidance data, the fingeringsomatosensory guidance data is sent to the somatosensory device, and thesomatosensory device performs a fingering guidance according to thefingering somatosensory guidance data.

In an implementable embodiment, when the operation instruction is theevaluation instruction, the virtual data includes a string key positionaccuracy, a fingering accuracy, and a rhythm sense. Currently, thespecific steps of executing the operation instruction to generate avirtual data are:

acquiring an operation data according to the user's actions, andextracting corresponding music score data based on the operation data,the music score data including note data, fingering data, and apre-defined standard interval duration between two notes;during the user's playing process, collecting, and recognizing the sounddata, obtaining tone data and volume data, and at the same timecollecting pressure data through a somatosensory device, and identifyingthe pressure data to obtain playing fingering data;matching the tone data and volume data with the note data, andgenerating string key position accuracy according to the matchingresult;matching the playing fingering data with the fingering data, andgenerating fingering accuracy according to the matching result;calculating a time difference between the pre-defined two notesaccording to the tone data, obtaining an actual interval duration, andgenerate a sense of rhythm according to the actual interval duration andthe standard interval duration.

In an implementable embodiment, when the operation instruction is anevaluation instruction, the virtual data includes string key positionaccuracy, fingering accuracy, and rhythm sense. Currently, afterexecuting the operation instruction to generate the virtual data, italso includes feedback guided steps, the specific steps are:

getting string key position accuracy, fingering accuracy, and rhythmsense;comparing the string key position accuracy with the pre-defined stringkey position accuracy threshold, comparing the fingering accuracy withthe pre-defined fingering accuracy threshold, and comparing the rhythmsense with the pre-defined rhythm sense threshold, and getting thecomparison results;when the fingering accuracy is less than the pre-defined fingeringaccuracy threshold, extract fingering practice score data;when the string key position accuracy is less than the pre-definedstring key position accuracy threshold, and the fingering accuracy isless than the pre-defined fingering accuracy threshold, extract a wrongparagraph and slow down the playback speed to practice;when the string key position accuracy is less than the pre-definedstring key position accuracy threshold, and the fingering accuracy isgreater than or equal to the pre-defined fingering accuracy threshold,extract the wrong paragraph, practice at normal speed first, and thenincrease the playback speed to practice;when the rhythm sense is greater than the pre-defined rhythm sensethreshold, the rhythm sense practice score data corresponding to therhythm sense practice rule is extracted according to the pre-definedrhythm sense practice rule.

Since the present invention adopts the above technical solutions, it hassignificant technical effects:

1. The present invention uses AR technology to improve the user'simmersion, and can be used for auxiliary teaching directly on Newguzheng. Compared with the existing technology, the portability is high;in the present invention, the virtual data is superimposed and displayedbased on the string distribution data, which can be used to guide thestrings without the need for a light bar indicator device, so the userdoes not need to re-tune after the installation of the system iscompleted, which can improve the user experience compared with the priorart;2. By designing the first pressure sensor and the second pressure sensorin the somatosensory device, the present invention can collect pressuredata for determining the force surface of the corresponding pick, sothat the data processing device can analyze the pressure data and theuser's fingering principles, and monitor the accuracy of the user'sfingering;3. The present invention can locate the position of each string throughthe design of the positioning devices for key position, therebydetermining the distribution of strings, and can also obtain the playingarea of the corresponding string according to the position of thestring; The design of the notch on the fixture is convenient for usersto install the positioning devices for key position.4. By designing the cloud server, the present invention enables eachuser's data processing device to interact through the cloud server,thereby providing users with functions such as multi-person coordinationand remote tutoring according to the user's needs.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to explain the embodiments of the present invention or thetechnical solutions more clearly in the prior art, the following willbriefly introduce the drawings required in the embodiments or thedescription of the prior art. Obviously, the drawings in the followingdescription are only these are some embodiments of the presentinvention. For those of ordinary skill in the art, without paying anycreative labor, other drawings can also be obtained based on thesedrawings.

FIG. 1 is a schematic diagram of module connections of an AR-basedsupplementary teaching system for guzheng of the present invention;

FIG. 2 is a schematic structural view of the key position positioningdevice 300 in Embodiment 1;

FIG. 3 is a schematic view of a use state of the positioning device forkey position 300 in FIG. 2;

FIG. 4 is a schematic diagram of module connections of the guzhengperipheral device 200 in FIG. 1;

FIG. 5 is a schematic structural diagram of the somatosensory device 400in FIG. 1;

FIG. 6 is a first explosion schematic diagram of the somatosensory smartfinger of FIG. 5;

FIG. 7 is a second explosion schematic diagram of the somatosensorysmart finger of FIG. 5;

FIG. 8 is a schematic diagram of the module connections of thesomatosensory smart finger in FIG. 5;

FIG. 9 is a schematic diagram of the workflow of an AR-basedsupplementary teaching method for guzheng of the present invention;

FIG. 10 is a schematic diagram of string guidance by dynamic points andstatic points in Embodiment 2.

In these figures:

-   100 represents AR device, 200 represents guzheng peripheral device,    210 represents sound input device, 220 represents data processing    device, 221 represents transmission module, 222 represents    identification module, 223 represents storage module, 224 represents    processing module, 300 represents positioning device for key    position, 310 represents identification panel, 320 represents    connection member, 330 represents fixing member, 340 represents    anti-skid sheet, 400 represents somatosensory device, 410 represents    wristband, 411 represents control module, 412 represents    somatosensory transmission module, 420 represents smart fingers, 421    represents pick installation sleeve, 422 represents first vibration    feedback device, 423 represents second vibration feedback device,    424 represents first pressure sensor, 425 represents second pressure    sensor, 426 represents first trigger, 427 represents second trigger,    500 represents cloud server.

DETAILED DESCRIPTION OF THE INVENTION

The present invention will be further described in detail below inconjunction with embodiments.

The following embodiments are explanations of the present invention andthe present invention is not limited to the following embodiments.

Embodiment 1

An AR-based supplementary teaching system for guzheng (hereinafterreferred to as a teaching system), as shown in FIG. 1, includes an ARdevice 100, a guzheng peripheral device No. 200, and several positioningdevices for key positions 300, the guzheng peripheral device 200 issignally connected to the AR device 100, the positioning devices for keyposition 300 is installed on the guzheng code, and the positioningdevices for key position 300 corresponds to the guzheng code one-to-one;

In this embodiment, the guzheng peripheral device 200 includes asignally connected sound input device 210 and a data processing device220, wherein the data processing device 220 is signally connected to theAR device 100.

The AR device 100 is used to obtain a real scene data.

The data processing device 220 is used to identify the guzheng and thepositioning devices for key position 300 in the real scene data togenerate a string distribution data; it is also used to obtain anoperation instruction according to user actions, executing the operationinstruction and generate a virtual data;

The AR device 100 is further configured to obtain the virtual data, andsuperimpose and display the virtual data with the real scene data basedon the string distribution data.

In this embodiment, the AR device 100 may use existing AR glasses or ahead-mounted AR display device;

This embodiment uses AR technology to improve user immersion. In thisembodiment, the virtual data is superimposed and displayed based on thestring distribution data, which can be used to guide the strings withoutthe need for a light bar indicator device, so the user does not need tore-tune after the installation of the system is completed, which canimprove the user experience compared with the prior art

Due to the different shapes of the guzheng, the different playing areasof each guzheng, and the thin strings, the identification accuracy ofdirect string identification is low, so it is not possible to directlyuse AR technology to superimpose virtual scenes. If there is a deviationbetween the virtual scene and the real scene, it cannot assist the userto practice. For this problem, in this embodiment, the positioningdevices for key position 300 is used to solve the problem.

As shown in FIG. 2, the positioning devices for key position 300includes an identification panel 310, a connecting member 320 and afixing member 330, wherein the fixing member 330 is provided with anotch matching a shape of the guzheng code, and the fixing member 330 isinstalled on an upper end of the guzheng code through the notch duringpositioning (as shown in FIG. 3); the identification panel 310 isconnected with the fixing member 330 by a connecting member 320;

In this embodiment, the positioning devices for key position 300 furtherincludes a non-slip sheet 340, which is installed on a contact surface(notch) of the fixing member 330 and the guzheng code, which enhances afriction between the fixing member 330 and the guzheng code, andeffectively prevents fixing member 330 from slipped off the guzhengcode;

In this embodiment, the identification panel 310 is rotatably connectedto the connection member 320, and the connection member 320 is rotatablyconnected to the fixing member 330, so that the tilt angle of theidentification panel 310 can be freely adjusted according to the actualsituation, which is convenient for the subsequent data processing device220 to quickly and accurately identify the identification panel 310based on the real scene image picked by the AR device 100.

Note: The identification panel 310, the connecting member 320, and thefixing member 330 are not in connected with the strings.

As can be seen from FIG. 3, in this embodiment, the positioning devicesfor key position 300 is placed on the guzheng code. The user only needsto simply install the fixing member 330 on the guzheng code. The userdoes not need to re-tune the guzheng, which is convenient to use; inthis embodiment, the position of the strings can be further determinedby identifying the identification panel 310, and the identificationpanel 310 is located at the guzheng code. At this time, the playing areaof the corresponding string can be determined according to theidentification panel 310 (the right side of the guzheng code, the smallhead side of the guzheng).

This embodiment combines the tracking registration technology based onmarkers and the tracking registration technology based on naturalfeatures to realize the identification and positioning of strings andcodes. The specific implementation method is as follows:

firstly, extracting the corresponding string feature points set from thepre-defined guzheng template image, and then extracting thecorresponding string feature points set from each frame of real scenedata acquired by the AR device 100, and track the spatial poses of theAR device 100 by matching the feature points sets to each other tocomplete a tracking registration of the guzheng strings; the featurepoint detection and feature matching methods are existing technologies,so they are not described in detail in this embodiment.

The positioning device for key position 300 is placed at each guzhengcode in advance, and the identification panel 310 displays a marker withthe content of the marker. The data processing device 220 recognizes themarker in the real scene data collected by the AR device 100 and obtainsthe fixed-point information of the markers, and then reconstructs thepredefined marker coordinates to the real scene marker coordinatesaccording to the principle of affine invariance of the graph Theposition change matrix to complete the tracking and registration of theguzheng code.

The tracking registration results of the obtained strings are screenedbased on the tracking registration results of the guzheng codes, thatis, the strings passing through the guzheng codes are extracted, and thestring distribution data is generated according to the extractionresults and the tracking registration results of the guzheng codes.

As shown in FIG. 4, the data processing device 220 includes atransmission module 221, an identification module 222, a storage module223, and a processing module 224, wherein the identification module 222and the sound input device 210, the transmission module 221, the storagemodule 223, and the processing module are respectively signallyconnected, and the processing module 224 is also connected to thetransmission module 221 and the storage module 223 respectively, and thetransmission module 221 is connected to the AR device 100;

The data processing device 220 is used to process the real scene datacollected by the AR device 100, specifically:

The transmission module 221 receives the real scene data and sends it tothe identification module 222;

The identification module 222 receives and identifies the real scenedata to obtain a string distribution data.

The data processing device 220 is also used to process a sound datacollected by the sound input device 210, specifically:

The voice input device 210 is used to collect the sound data when theuser plays and send the sound date to the identification module 222;

The identification module 222 is configured to receive and identify thesound data and obtain a tone data;

The processing module 224 is configured to retrieve the operationinstruction associated with the tone data from the storage module 223,execute the operation instruction, and generate virtual data accordingto the operation instruction.

The data processing device 220 is also used to generate virtual databased on the tone data and string distribution data and control adisplay of the virtual data, specifically:

The processing module 224 retrieves the operation instruction associatedwith the tone data from the storage module 223, executes the operationinstruction, generates virtual data according to the operationinstruction, and sends the virtual data to the AR device 100 through thetransmission module 221, and controls the AR device 100 to superimposeand display the virtual data and the real scene data.

The operation instruction include playing guidance instruction andevaluation instruction.

In this embodiment, the storage module 223 is used to store all dataused to supplementary teaching, such as music score data, basic musictheory data, association rule data, and practice music score data.

The music score data includes fingering data, note data and timeinterval data of adjacent notes (standard time interval data between twoadjacent notes);

basic music theory data includes pre-stored guzheng tone and fingering3D animation;the association rule data includes pre-defined operation associationrules, pre-defined string association rules and pre-defined fingeringassociation rules;the practice music score data includes fingering practice score data andrhythm practice score data;

In actual use, the identification module 222 matches a received soundinformation with the pre-stored guzheng tones, obtains a matched guzhengtones, uses the guzheng tones as the tone data, and extracts the stringcode associated with the guzheng tones and musical notes.

The processing module 224 can determine and respond to the user'soperation according to the association relationship between a stringencoding and the operation data, to obtain a corresponding operationinstruction.

In this embodiment, string guidance data, fingering guidance data,and/or numbered musical notation display data can be generated accordingto the operation instruction, and superimposed and displayed by the ARdevice 100, specifically:

the processing module 224 obtains operation data according to the user'sactions, and extracts corresponding music score data based on theoperation data;when the numbered musical notation is displayed, the processing module224 retrieves the corresponding numbered musical notation display datafrom the storage module 223, and controls the AR device 100 tosuperimpose the numbered musical notation display data on top of theguzheng based on the guzheng string distribution data, so that the ARdevice 100 displays a corresponding numbered musical notation for theusers.when performing the string guidance, according to the note data in thescore data, the processing module 224 acquires a string numberassociated with the note data and generates string guidance data, thestring guidance data includes a string position data (string number), avirtual prompt data (a symbol used to prompt the user to play thestring) and the time interval data of adjacent notes;

The AR device 100 superimposes the virtual prompt data to thecorresponding strings according to the string position data based on theguzheng string distribution data and the time interval data of adjacentnotes, achieving string guidance.

When performing fingering guidance, according to the fingering data inthe score data, the processing module 224 obtains the fingeringthree-dimensional animation associated with the fingering data, andgenerates fingering guidance data in combination with theabove-mentioned string number, the fingering guidance data includesstring position data (string number), fingering three-dimensionalanimation and time interval data of adjacent notes;

In this embodiment, based on the guzheng string distribution data andthe time interval data of adjacent notes, the AR device 100 superimposesthe three-dimensional fingering animation on the corresponding stringaccording to the string position data, to realize the common guidance ofthe string and fingering.

For example, after the user selects a certain score from the music scorelist by moving the strings, the processing module 224 extracts thefingering data, the note data, the time interval data of adjacent notes,and numbered musical notation display data corresponding to the selectedscore from the storage module 223, obtains string position datacorresponding to each note based on the note data, acquires athree-dimensional fingering animation corresponding to a fingering basedon the fingering data, and generates string guidance data, fingeringguidance data, and musical notation display data as the virtual data;

The AR device 100 receives the above virtual data and performssuperimposed display based on the string distribution data. At thistime, the numbered musical notation display data is located above theguzheng. The three-dimensional fingering animation and the virtualprompt data for guiding the strings are superimposed on thecorresponding strings. The user can follow the numbered musical notationdisplay data and the virtual prompt data and imitate the correspondingfingering three-dimensional animation to learn guzheng playing.

During the entire playing process, the identification module 222collects the sound data played by the user in real time through thesound input device 210, matches the obtained sound data with theprestored guzheng tone in the storage module 223, and obtains tone data;the processing module 224 matches the tone data with the note data ofthe corresponding music score to locate a current playing progress, andupdates the string guidance data, the fingering guidance data, and thenotation display data based on the current playing progress.

Note: The storage module 223 can also store virtual scene data andvirtual game data;

The processing module 224 obtains an operation data according to user'sactions, extracts the virtual scene data selected by the user from thestorage module 223 according to the operation data, and sends thevirtual scene data to the AR device 100. The AR device 100 superimposesthe virtual scene data based on the string distribution data outside theguzheng, that is, to provide the user with a virtual playing environment(such as natural environment, stage, etc.) outside the guzheng.

The processing module 224 obtains the operation data according to user'sactions, extracts virtual game data selected by the user from thestorage module 223 according to the operation data, and sends thevirtual game data to the AR device 100. The AR device 100 superimposesthe virtual game data based on the string distribution data at theguzheng, that is, to provide the user with an interactive game ofguzheng playing.

In another embodiment, the teaching system further includes asomatosensory device 400, and the somatosensory device 400 is signallyconnected to the transmission module 221;

The processing module 224 is further configured to generate fingeringsomatosensory guidance data according to the operation instruction, andsend the fingering somatosensory guidance data to the somatosensorydevice 400 through the transmission module 221 to control thesomatosensory device 400 to perform somatosensory feedback according tothe somatosensory guidance data.

In this embodiment, the storage module 223 stores the somatosensoryfeedback data corresponding to fingerings one by one, and the processingmodule 224 extracts a corresponding somatosensory feedback data from thestorage module 223 based on the fingering data from the acquired musicscore data to generate fingerings somatosensory guidance data. Thefingering somatosensory guidance data includes somatosensory feedbackdata and time interval data of adjacent notes;

According to the somatosensory feedback data, the somatosensory device400 performs a vibration prompting based on the time interval data ofadjacent notes. The somatosensory feedback and the above-mentionedthree-dimensional fingering animation can be individually/commonlydirected.

In this embodiment, as shown in FIG. 5, the somatosensory device 400includes a wristband 410 and four somatosensory smart fingers. Eachsomatosensory smart finger includes a smart finger 420, a pickinstallation sleeve 421, a first vibration feedback device 422 and asecond vibration feedback device 423;

The specific structure of the somatosensory smart fingers is shown inFIGS. 6 and 7. The smart fingers 420 is provided with a pickinstallation sleeve 421, a first vibration feedback device 422 and asecond vibration feedback device 423, wherein all the first vibrationfeedback devices 422 are located on a same side of the pick installationsleeve 421, all the second vibration feedback devices 423 are located onan opposite side of the pick installation sleeve 421, and the pickinstallation sleeve 421 is used to install the pick;

The wristband 410 is provided with a somatosensory transmission module412, a battery module, and a control module 411, wherein the batterymodule supplies power to the first vibration feedback device 422, thesecond vibration feedback device 423, the somatosensory transmissionmodule 412 and the control module 411, that is, to power thesomatosensory device 400. The control module 411 is signally connectedto the first vibration feedback device 422, the second vibrationfeedback device 423, and the somatosensory transmission module 412respectively, and the somatosensory transmission module 412 is signallyconnected to the transmission module 221;

The control module 411 is used to receive fingering somatosensoryguidance data through the somatosensory transmission module 412 andcontrol the first vibration feedback device 422 and the second vibrationfeedbacker 423 to perform a vibration feedback according to thefingering somatosensory guidance data.

In actual use, the user can insert a pick into the pick installationsleeve 421, and then use the corresponding fingering to move the stringsbased on the vibration feedback performed by the first vibrationfeedback device 422 and the second vibration feedback device 423;

In this embodiment, x1 represents the first vibration feedback device422, and x2 represents the second vibration feedback device 423;

x includes a, b, c, and d, wherein a represents a thumb, b represents anindex finger, c represents a middle finger, and d represents a ringfinger;

Each fingering specifically corresponds to a vibration area of avibration feedback device on the somatosensory device 400, such as: hold(

in Chinese), a2; split (

in Chinese), a1; wipe (

in Chinese), b2; pick (

in Chinese), b1; hook (

in Chinese), c2; tick (

in Chinese), c1; lift (

in Chinese), d1; flick (

in Chinese), alternate response of a1 and a2; or alternate response ofb1 and b2.

In this embodiment, an upper mounting member (shown as a circle in FIG.6), a first pressure sensor 424, and a second pressure sensor 425 (asshown in FIG. 7) are provided on the lower surface of the pickinstallation sleeve 421. The first pressure sensor 424 is located on theside away from the pick, the second pressure sensor 425 is located on aside near the pick, and the first pressure sensor 424 and the secondpressure sensor 425 are both signally connected to the control module411;

An upper surface each of the smart fingers 420 is provided with a lowermounting member (as shown by a circle in FIG. 6), a first trigger 426and a second trigger 427 (as shown in FIG. 6). A first trigger 426corresponds to the first pressure sensor 424, the second trigger 427corresponds to the second pressure sensor 425, and the lower mountingmember is rotatably connected to the upper mounting member, so that whena pick move the strings, the first pressure sensor 424 squeeze the firsttrigger 426, or the second pressure sensor 425 squeeze the secondtrigger 427;

Note: The upper mounting member and the lower mounting member are usedto install the pick installation sleeve 421 at the smart finger 420; andwhen the pick moves the strings, the first pressure sensor 424 cansqueeze the first trigger 426 or the second pressure sensor 425 cansqueeze the second trigger 427;

The control module 411 receives a pressure data detected by the firstpressure sensor 424 and the second pressure sensor 425, and sends theobtained pressure data to the data processing device 220 through thesomatosensory transmission module 412. The data processing device 220analyzes a fingering accuracy according to the pressure data.

In this embodiment, the pressure data detected by the first pressuresensor 424 and the second pressure sensor 425 can be used to determine aplucking situation of the pick, such as when a front end of the pickpresses the strings (an lower surface of the pick is subjected to anupward force), a tail end of the pick will be depressed, at this timethe first pressure sensor 424 will squeeze the first trigger 426, so asto determine the corresponding fingering according to the pressure dataof the first pressure sensor 424 and the smart finger 420 on which it islocated; Similarly, when picking strings by the front of the pick (theupper surface of the pick is subjected to an downward force), the tailend of the pick will be pressed upward, and the second pressure sensor425 will squeeze the second trigger to determine the correspondingfingering according to the pressure data of the second pressure sensor425 and the smart finger 420 on which it is located.

When the pick move the strings, the pick installation sleeve 421 rotatesslightly, so the first trigger 426 and the second trigger 427 in thisembodiment are designed are both in a convex structure, so that apressure value of the first pressure sensor 424/the second pressuresensor 425 is exactly 0 or there is an initial pressure value when thepick is not being played, so that when the pick moves the strings, thefirst pressure sensor 424/the second pressure sensor 425 can detect achange in pressure. The control module 411 collects a pressure datathrough the first pressure sensor 424 and the second pressure sensor425, and sends the pressure data to the transmission module 221 throughthe somatosensory transmission module 412 (the pressure data includes asmart fingers number, a pressure sensor number and a pressure value);

The identification module 222 includes an image identification unit, asound identification unit, and a somatosensory identification unit thatare connected to the processing module 224 and the storage module 223.The image identification unit and the somatosensory identification unitare both signally connected to the transmission module 221, and thesound identification unit and the sound input device 210 are signallyconnected.

The image identification unit is used to recognize the guzheng and thepositioning devices for key position 300 in a real scene data to obtaina string distribution data;

The voice identification unit is used to recognize the sound datacollected by the voice input device 210 to obtain a corresponding tonedata;

The somatosensory identification unit is used to recognize the pressuredata and obtain corresponding playing fingering data.

In actual use: the first pressure sensor 424 represented by X1, and thesecond pressure sensor 425 represented by X2;

X includes A, B, C, and D, where A is the thumb, B is the index finger,C is the middle finger, and D is the ring finger;Hold, A2; Split, A1; Wipe, B2; Pick, B1; Hook, C2; Tick, C1; Lift, D1;Flick, A1A2 alternate response or B1B2 alternate response. Suchfingering identification only needs to be judged and recognized by thepressure sensor of the feedback glove.

Continuous or double hold, A2 continuous response; continuous or doublewipe, B2 continuous response; continuous or double hook, C2 continuousresponse; such fingering identification requires finger identificationthrough the combination of somatosensory device 400 and voiceidentification, through the body sensory device 400 judges the fingeringif is a combination of fingering, and through the sound recognition unitjudges a difference between the tones, with the same tones being acontinuous holds, continuous wipes or continuous hooks, and thedifferences in tone are double holds, double wipes or double hooks.

In another embodiment, the teaching system further includes a cloudserver 500. The cloud server 500 is respectively connected to thetransmission modules 221 of a plurality of data processing devices 220.The data processing device 220 is connected to other data processingdevices through the cloud server 500 to perform a data interaction.

The operation instruction also include multi-player collaborativeperformance and instructor remote guidance;

When the operation instruction is a multi-player collaborativeperformance/instructor remote guidance, the processing module 224 inputsan video data of the user's performance through an external cameradevice, and also uses the sound input device 210 input such as a sounddata of the user's performance, the obtained video data and sound dataare synchronized to the cloud server 500 through the transmission module221, and downloads the video data and sound data of the correspondingcompanion (coordinator/instructor/student) from the cloud server 500;the processing module 224 processes the obtained video data to obtainvirtual video data, and each audio data is mixed to obtain a mixed audiodata, the virtual video data is displayed through the AR device 100, andthe mixed audio data is played through the AR device 100 or theperipheral sound device.

When the instructor remotely guides, the processing module 224 alsocollects the user's somatosensory data through the somatosensory device400, synchronizes it to the cloud server 500 through the transmissionmodule 221, and synchronously downloads the corresponding instructor'ssomatosensory data from the cloud server 500. A learner can not onlyfeel the guidance of remote instructor through images and sounds, butalso feel the real-time fingering and somatosensory response of remoteinstructor, to realize a more realistic remote guzheng teaching.

Embodiment 2

An AR-based supplementary teaching method for guzheng implemented basedon the system described in Embodiment 1, as shown in FIG. 9, includesthe following steps:

S100: set a positioning device for key position 300 on each guzheng codeof guzheng;

S200: obtain a real scene data through the AR device 100, identify theguzheng and the positioning devices for key positions in the real scenedata, and generate the string distribution data; S300: display anoperation interface for a user through the AR device 100, at the sametime obtain an operation instruction according to a user's actions,execute the operation instruction and generate the virtual data, sendthe virtual data to the AR device 100, and the AR device 100superimposes and displays the virtual data with the real scene databased on the string distribution data.

This embodiment uses AR technology to improve the user's immersion. Inthis embodiment, a positioning device for key position 300 is providedon the guzheng code, so that the user does not need to retune theguzheng, which is convenient to use.

The teaching of guzheng is different from the learning of ordinaryknowledge. The textbook knowledge cannot be simply displayed to studentsthrough AR technology, but the teaching of guzheng in real scenes needsto be taught. In reality, the shape of each guzheng is different. Thepositioning devices for key position 300 realizes an positioning of theguzheng strings to generate string distribution data, so that the ARdevice 100 superimposes and displays the virtual data based on thestring distribution data, thereby being able to show the user a virtualscene corresponding to the guzheng.

In step S200, the specific method of generating string distribution datais as follows:

recognize the guzheng in the real scene data, obtain a guzhengmorphological data, and identify the guzheng surface and strings in theguzheng morphological data;recognize the positioning devices for key position 300 in the guzhengmorphological data, obtain a positioning points, obtain the number andan arrangement order of the positioning devices for key positions 300according to the positioning point, and determine the guzheng tonecorresponding to each positioning point according to the number andarrangement order, and encode and record the strings passing throughthis positioning point to generate a string distribution data.

As can be seen from the above, the guzheng tone and the correspondingstring position are bound through the positioning points, therebygenerating string association rules.

Note that the guzheng tone is also associated with each note in advance.

In this embodiment, firstly, the overall morphological of guzheng isdistinguished from the external environment area based on the existingimage identification technology, and secondly, the surface and stringsin the overall morphological of guzheng are recognized; because thestrings are thin, the accuracy of direct identification is low, andcannot identify the playing area of the guzheng, so in this embodiment,a positioning devices for key position 300 is provided on the guzhengcode, the key string positioning device 300 is recognized to realize aconfirmation of the string, and the positioning devices for key position300 is located at on the guzheng code, the playing area of the stringscan be determined according to the identified positioning point and thecorresponding strings.

In S300, the AR device 100 is used to display an operation interface forthe user, and the specific steps for obtaining the operation instructionaccording to the user's actions are:

The AR device 100 displays an operation interface for the user, andsimultaneously collects and recognizes sound data to obtain tone data;obtains operation data corresponding to the tone data according to apre-defined operation association rule, and generates operationinstruction based on the operation data;

The operation instruction includes playing guidance instruction andevaluation instruction.

Since different strings of guzheng correspond to different tones, thestrings are mapped with operation data and tone in advance to generateoperation association rules. The operation data includesupward/leftward, downward/rightward, cancellation, confirmation, etc.;The user selects a desired service by moving the strings. At this time,a corresponding operation instruction is obtained according to theuser's choice;

In actual use, the AR device 100 displays an operation interface for theuser, and the operation interface displays a list of services (playingguidance service and evaluation service) and operation instruction (themapping relationship between strings and operation data), such as thevirtual icons representing the operation data are superimposed anddisplayed on the strings, that is, icons such as upward/leftward,downward/rightward, cancellation, confirmation are superimposed on thecorresponding strings.

The user can move the corresponding strings according to actual needs.At this time, the sound data is collected and recognized to obtain thetone data, and the user's operation is identified according to thepre-defined operation association rules, the service selected by theuser is obtained, and the corresponding operation instruction isgenerated.

S310. When the operation instruction is a playing guidance instruction,the virtual data includes string guidance data, fingering guidance data,and/or numbered musical notation display data, and the specific stepsfor generating virtual data by executing the operation instruction atthis time are:

acquire an operation data according to the user's actions, andextracting corresponding music score data based on the operation data,the music score data including note data and fingering data;generating string guidance data, fingering guidance data and/or numberedmusical notation display data according to the playing guidanceinstruction, including:extract the corresponding string position according to the note databased on the pre-defined string association rules, and generate a stringguidance data;based on the pre-defined fingering association rules, extract acorresponding fingering 3D animation according to the fingering data togenerate a fingering display guidance data;extract the musical notation display data corresponding to the musicscore data. In the actual operation process, a music score list isextracted according to the playing guidance instruction, and the musicscore list is displayed through the AR device 100. At this time, theuser moves the strings to select the music score from the music scorelist to obtain a music score information;the playing guidance instruction includes at least one of a scoredisplay command, a string guidance command, and a fingering guidancecommand;

A. When the play guidance instruction includes a score display command:

Extracting the musical notation display data and note data correspondingto the music score data, displaying the musical notation display datathrough the AR device 100, and at the same time collecting andrecognizing sound data in real time to obtain tone data, according tothe matching of the tone data and the note data, generating music scoreprogress positioning data, updating a displayed musical notation displaydata according to the music score progress data; this embodiment can notonly display the music notation corresponding to the music score for theuser through AR technology, but also recognize the sound data played bythe user, the automatic line-wrapping/page-turning of the numberedmusical notation based on the sound data. Note that the tone data ismatched with the note data, and generating music score progresspositioning data is a prior art, so it will not be described in detailin this specification.

Since the beginners of guzheng have a weak sense of rhythm and areunskilled in playing, such as automatic line-wrapping/page-turning ofthe numbered musical notation according to standard musical notationplayback data, users often cannot keep up with the playback progress ofmusical notation. Based the sound data played by the user in thisembodiment can realize the automatic line-breaking/page-turning of themusical notation, which can improve the user's experience and meet theuser's needs.

B. When the play guidance instruction includes a string guidanceinstruction:

Extracting the note data corresponding to the music score data,extracting corresponding string positions according to the note databased on pre-defined string association rules, and generating stringguidance data;

In this embodiment, string guidance data is generated according to thebar, that is, the string guidance data includes virtual prompt data,time interval of adjacent notes, and string position data; According tothe string guidance data, the AR device 100 superimposes the virtualprompt data on the strings to be played in sequence based on the timeinterval data of adjacent notes. At this time, the user can see thestrings corresponding to the music score one by one through the ARdevice 100, and then the user can play following the fingering accordingto the key position of the string that the user sees.

Those skilled in the related field can set the style of the virtualprompt data according to actual needs, such as dots and lines, and thoseskilled in the related art can also set the guide mode according to theactual needs. This embodiment includes the following two guide modes:

Guide by bars: At this time, the virtual prompt data is linear, which isthe same length as the playing area of the corresponding string, and thevirtual prompt data is sequentially superimposed on the correspondingstring according to the note data and string position data for display,so continuously demonstrate the key information of a string for theuser, and then the user plays it according to the music score;

Guide by strings: At this time, the virtual prompt data includes dynamicpoints and static points (as shown in FIG. 10), and the static pointsidentify the best playing points of the corresponding strings (themethod of determining the best playing points is the existingtechnology, so no more details in this specification), the dynamic pointmoves along the corresponding string to the static point, and themovement duration is used to reflect its playing rhythm, that is, whenthe dynamic point and the static point are superimposed on a certainstring for displaying, indicating that the string is the next string tobe played. When the dynamic point moves to the static point, itindicates that this string needs to be currently played;

It can be seen from the above that the design of the string guidancedata in this embodiment can guide the user to the key position of thestring to be played according to the music score, and in thisembodiment, by designing the dynamic point and the static point, theuser is guided to the string at the same time, it can also guide theuser's playing rhythm.

In this embodiment, the string guidance data may also include an errorfeedback data, that is, if the user still plays a wrong key positionduring the playing process according to the guidance of the virtualprompt data, or does not play when the dynamic point and the staticpoint coincide, At this time, the error feedback data is superimposed onthe corresponding string for display. The error feedback data can belinear (the color is different from the virtual prompt data), and canalso be “!”, “X” and other signs, this embodiment does not make specificrestrictions.

The method of judging whether the user's string key position is correctis:

collecting the sound data during the user's playing process, identifyingthe sound data, obtaining the tone data, and matching the tone data withthe note data, when the match is successful, it is judged that thestring key is played correctly, otherwise it is judged as error.

The method to judge whether the user plays the rhythm correctly is:

extracting the standard interval time between adjacent notes and theactual playing interval time of the user. When the absolute value of thedifference between the standard interval time and the playing intervaltime is greater than the pre-defined interval threshold, it isdetermined that the playing rhythm is wrong.

The standard interval time is the time taken for the dynamic point tomove to the static point, and a person skilled in the related art canset the interval threshold according to actual needs.

C. When the playing guidance instruction includes a fingeringinstruction command, there are two ways to perform fingering instructionbased on AR display and fingering instruction based on somatosensoryfeedback.

C1. Fingering guidance based on AR display:

extract the fingering data corresponding to the music score data,extract the corresponding fingering three-dimensional animationaccording to the fingering data based on pre-defined fingeringassociation rules, and generate fingering display guidance data;fingering association rules are used to associate fingering names withcorresponding fingering 3D animations;the fingering data refers to the respective fingerings in the process ofplaying the corresponding score. In this embodiment, the correspondingfingering three-dimensional animations are sequentially extractedaccording to the fingering data, and the obtained fingeringthree-dimensional animations are displayed by the AR device 100;

In this embodiment, the fingering instruction command may be associatedwith the above step A and/or step B, so that the three-dimensionalfingering animation is switched according to the sound data played bythe user (in this case, the note data corresponds to the fingering dataone-to-one).

That is, the user recognizes the sound data played by the user to obtaintone data, matches the tone data and the note data, and generates musicscore progress positioning data, and switches the displayedthree-dimensional fingering animation based on the music score progresspositioning data.

In this embodiment, the fingering instruction command may be associatedwith step B, so that fingering guidance is performed at the same time asstring guidance (at this time, note data corresponds to fingering dataone-to-one).

That is, based on the string position data, the obtainedthree-dimensional fingering animation is superimposed on thecorresponding string for guidance.

As can be seen from the above, this implementation provides users withthree-dimensional dynamic fingering instruction through AR technology,which is convenient for users to accurately understand the fingeringpoints.

C2. Fingering guidance based on somatosensory feedback:

Based on the pre-defined fingering association rules, correspondingsomatosensory feedback data is extracted according to the fingeringdata, fingering somatosensory guidance data are generated, and thefingering somatosensory guidance data is sent to the somatosensorydevice 400, and the somatosensory device 400 is guided according to thefingering somatosensory guidance data. For specific guidance methods,see Embodiment 1. In this embodiment, the description will not berepeated.

S320. When the operation instruction is an evaluation instruction, thevirtual data includes string key position accuracy, fingering accuracy,and rhythm sense. At this time, the specific steps of executing theoperation instruction to generate a virtual data are:

S321. acquire an operation data according to the user's actions, andextract corresponding music score data based on the operation data. Themusic score data includes note data, fingering data, and a pre-definedstandard interval between two notes;

The pre-defined two notes include, but are not limited to, two adjacentnotes, the first note and the last note of the bar, the first note andthe last note of the entire song.

S322. During the user's playing process, collect and recognize sounddata to obtain tone data, and at the same time collect pressure datathrough the somatosensory device 400, and identify the pressure data toobtain playing fingering data;

The pressure data is used to determine which side of the pick is to movethe string, to determine the playing fingering data for playing based onthe fingering principle;

S323: Match the tone data with the note data, and generate string keyposition accuracy according to the matching result;

That is, calculate a proportion of the correct notes played in theentire song to the total notes, and use the results as the string keyposition accuracy;

S324. Match the playing fingering data with the fingering data, andgenerate fingering accuracy according to the matching result;

That is, calculate the proportion of correct fingering played in thewhole song to the total fingering, and use the result as fingeringaccuracy;

S325. Calculate the time difference between the pre-defined two notesaccording to the tone data, obtain the actual interval duration, andgenerate a sense of rhythm according to the actual interval duration andthe standard interval duration.

In this embodiment, the whole song is taken as an example, that is, twonotes are pre-defined as the first note and the last note of the musicscore, and the standard interval length is extracted, that is, thestandard length of the entire song, and the total number of times theuser plays the entire song is recorded. Use time as the actual intervalduration, and calculate the absolute value of the difference between theactual interval duration and the standard interval duration. The largerthe absolute value obtained, the worse the sense of rhythm;

In this embodiment, the ratio of the absolute value of the differencebetween the actual interval duration and the standard interval durationto the standard interval duration is used as the characteristic value ofthe sense of rhythm.

In this embodiment, when the operation instruction is an evaluationinstruction, the virtual data includes string key position accuracy,fingering accuracy, and rhythm sense. At this time, after executing theoperation instruction to generate virtual data, a feedback guidance stepis further included, the specific steps are:

a. Obtain the string key position accuracy, fingering accuracy, andrhythm;b. Compare the string key position accuracy with the pre-defined stringkey position accuracy threshold, compare the fingering accuracy with thepre-defined fingering accuracy threshold, and compare the rhythm withthe pre-defined rhythm threshold, and get the comparison result;

The above-mentioned string key position accuracy threshold, fingeringaccuracy threshold and rhythm threshold can be set according to actualconditions, and this embodiment does not limit them;

c. When the fingering accuracy is less than the pre-defined fingeringaccuracy threshold, extract fingering practice score data;

The fingering practice score data is an exercise score with samefingering but in different string positions. In this embodiment,repeated guidance exercises are performed according to the fingeringpractice score data to improve the user's fingering proficiency andaccuracy.

d. When the string key position accuracy is less than the pre-definedstring key position accuracy threshold, and the fingering accuracy isless than the pre-defined fingering accuracy threshold, extract thewrong paragraph and slow down the playback speed to practice;

When the accuracy of the strings is low and the accuracy of thefingering is also low, it means that the speed of the score is too fastand the user cannot learn effectively. Therefore, the wrong part ispicked out separately, and the playback speed is slowed down forrepeated guided practice.

e. When the string key position accuracy is less than the pre-definedstring key position accuracy threshold, and the fingering accuracy isgreater than or equal to the pre-defined fingering accuracy threshold,extract the erroneous section, practice at normal speed first, and thenincrease the playback speed to practice;

When the accuracy of the strings is low but the accuracy of fingering ishigh, it means that the user has the wrong string position to playinertia, so the wrong part is picked out separately, and repeatedpractice at the normal playback speed to correct it, and then acceleratethe playback speed to strengthen stringing habits.

f. When the rhythm sense is greater than the pre-defined rhythm sensethreshold, the rhythm sense practice score data corresponding to therhythm sense practice rule is extracted according to the pre-definedrhythm sense practice rule.

The pre-defined rhythm practice rule is that the rhythm sense is gradedin advance. If the data corresponding to the rhythm sense is greaterthan 50%, the rhythm sense is poor. At this time, simpler rhythmpractice score data is provided for users to practice, for example,scores for different string positions but the same fingering and rhythm;when the data corresponding to the sense of rhythm is greater than 20%and less than or equal to 50%, it means poor sense of rhythm. At thistime, complex rhythm practice score data is provided for users topractice, for example, scores for different string positions anddifferent fingerings, but the same rhythm for practicing.

A person skilled in the related art may grade the rhythm sense byhimself and set rhythm sense practice score data corresponding to eachlevel, which is not limited in this embodiment.

This embodiment can detect the accuracy of string keys and fingeringduring the user's playing, and can also calculate the user's sense ofrhythm, and feedback the calculation results to the user, so that theuser can understand the weak points in the learning of guzheng, and canalso generate corresponding practice music scores for users to practice.

The embodiments in this specification are described in a progressivemanner. Each embodiment focuses on the differences from otherembodiments, and the same or similar parts between the embodiments mayrefer to each other.

It should be noted:

The reference to “one embodiment” or “embodiment” in the specificationmeans that a feature, structure, or characteristic described inconnection with the embodiment is included in at least one embodiment ofthe present invention. Therefore, the phrases “one embodiment” or“embodiments” appearing in various places throughout the specificationdo not necessarily all refer to the same embodiment.

Although the preferred embodiments of the present invention have beendescribed, those skilled in the art can make additional changes andmodifications to these embodiments once they learn the basic inventiveconcept. Therefore, the appended claims are intended to be interpretedas including the preferred embodiments and all changes and modificationsfalling within the scope of the present invention.

In addition, it should be noted that the specific embodiments describedin this specification may have different parts, shapes of parts, namestaken, etc. Any equivalent or simple changes made according to thestructure, features and principles described in the patent concept ofthe present invention are included in the protection scope of thepresent invention. Those skilled in the art to which the presentinvention pertains can make various modifications or additions to thedescribed specific embodiments or substitute in a similar manner, aslong as they do not deviate from the structure of the present inventionor go beyond the scope defined by the claims, All should belong to theprotection scope of the present invention.

What is claimed is:
 1. An AR-based supplementary teaching system forguzheng, comprising an AR device, a data processing device, and aplurality of positioning devices for key positions, wherein the dataprocessing device is signally connected to the AR device, thepositioning devices for key positions are installed on a guzheng code,and each of the positioning devices for key positions correspondsone-to-one with the guzheng code; the AR device is used to obtain a realscene data; the data processing device is used to identify the guzhengand the positioning devices for key positions in the real scene data,generating a string distribution data; the data processing device isalso used to obtain an operation instruction based on user's actions,executing the operation instruction, and generating a virtual data; andthe AR device is also used to superimpose and display the virtual datawith the real scene data based on the string distribution data.
 2. TheAR-based supplementary teaching system for guzheng according to claim 1,wherein the system further comprises a sound input device, the soundinput device is signally connected to the data processing device; thevoice input device is used to collect a sound data when the user playsand send the sound data to the data processing device; the dataprocessing device is configured to identify the sound data, generate atone data, retrieve the operation instruction associated with the tonedata, execute the operation instruction, and generate the virtual data.3. The AR-based supplementary teaching system for guzheng according toclaim 2, wherein the data processing device comprises a transmissionmodule, an identification module, a storage module, and a processingmodule, wherein the identification module is signally connected to thesound input device, the transmission module, the storage module, and theprocessing module respectively, the processing module is also signallyconnected to the transmission module and the storage modulerespectively, and the transmission module is signally connected to theAR device; the identification module is used for receiving andidentifying the sound data to obtain the tone data, and also forreceiving and identifying the real scene data to obtain the stringdistribution data; the processing module is used for retrieving theoperation instruction associated with the tone data from the storagemodule, executing the operation instruction, and generating the virtualdata according to the operation instruction; and also for controllingthe AR device to superimpose and display the virtual data with the realscene data; the virtual data comprises a string guidance data, afingering guidance data, and a numbered musical notation display data.4. The AR-based supplementary teaching system for guzheng according toclaim 1, wherein the system further comprises a somatosensory device,the somatosensory device is signally connected to the data processingdevice; the data processing device also used to generate a fingeringsomatosensory guidance data according to the operation instruction, andcontrol the somatosensory device to perform a somatosensory feedbackaccording to the somatosensory guidance data.
 5. The AR-basedsupplementary teaching system for guzheng according to claim 4, whereinthe somatosensory device comprises a wristband and four smart fingers;each of the smart fingers is provided with a pick installation sleeve, afirst vibration feedback device and a second vibration feedback device,wherein all first vibration feedback devices are located on a same sideof the pick installation sleeves, all second vibration feedback devicesare located on an opposite side of the pick installation sleeves, andeach of the pick installation sleeves is used to install a pick; thewristband is provided with a somatosensory transmission module, abattery module, and a control module, wherein the battery modulesupplies power to the first vibration feedback devices, the secondvibration feedback devices, the somatosensory transmission module, andthe control module; the control module is signally connected to thefirst vibration feedback devices, the second vibration feedback devices,and the somatosensory transmission module respectively; and thesomatosensory transmission module is signally connected to thetransmission module; the control module is used for receiving fingeringsomatosensory guidance data by the somatosensory transmission module,and controlling the first vibration feedback devices and the secondvibration feedback devices to perform a vibration feedback according tothe fingering somatosensory guidance data.
 6. The AR-based supplementaryteaching system for guzheng according to claim 5, wherein a lowersurface of each of the pick installation sleeves is provided with anupper mounting member, a first pressure sensor, and a second pressuresensor; the first pressure sensor is located on a side away from thepick, and the second pressure sensor is located on a side close to thepick side, the first pressure sensor and the second pressure sensor areboth signally connected to the control module; wherein an upper surfaceof each of the smart fingers is provided with a lower mounting member, afirst trigger, and a second trigger; the first trigger corresponds tothe first pressure sensor, and the second trigger corresponds to thesecond pressure sensor; the lower mounting member and the upper mountingmember are rotatably connected, so that when the pick move the string,the first pressure sensor squeezes the first trigger or the secondpressure sensor squeezes the second trigger; wherein the control moduleis used to receive a pressure data detected by the first pressure sensoror the second pressure sensor, and send the obtained pressure data tothe data processing device by the somatosensory transmission module, andthe data processing device analysis a fingering accuracy according tothe pressure data.
 7. The AR-based supplementary teaching system forguzheng according to claim 1, wherein each of the positioning devicesfor key positions comprises an identification panel, a connecting memberand a fixing member; the fixing member is provided with a notch matchinga shape of the guzheng code, the fixing member is installed on an upperend of the guzheng code through the notch during positioning; and theidentification panel is connected to the fixing member through theconnecting member.
 8. The AR-based supplementary teaching system forguzheng according to claim 1, wherein the system further comprises acloud server; the cloud server is in signal connection with a pluralityof data processing devices, and each of the plurality of data processingdevices performs data interaction with other data processing devicesthrough the cloud server.
 9. An AR-based supplementary teaching methodis realized by using the system according to claim 1, wherein the methodcomprises the following steps: setting a positioning device for keypositions on each guzheng code of the guzheng; obtaining the real scenedata through the AR device, identifying the guzheng and the positioningdevices for key positions in the real scene data, and generating thestring distribution data; displaying an operation interface for the userby the AR device, obtaining the operation instruction according to theuser's actions simultaneously, executing the operation instruction andgenerating the virtual data, sending the virtual data to the AR device,and the AR device superimposing and displaying the virtual data with thereal scene data based on the string distribution data.
 10. The AR-basedsupplementary teaching method according to claim 9, wherein the step ofdisplaying an operation interface for the user by the AR device,obtaining the operation instruction according to the user's actionssimultaneously comprises: displaying an operation interface for the userby the AR device, and simultaneously collecting and identify a sounddata to obtain a tone data; obtaining an operation data corresponding tothe tone data according to a pre-defined operation association rule, andgenerating operation instruction based on the operation data; theoperation instruction comprises a playing guidance instruction and anevaluation instruction; when the operation instruction is the playingguidance instruction, the virtual data comprises a string guidance data,a fingering guidance data, and/or a numbered musical notation displaydata.
 11. The AR-based supplementary teaching system for guzhengaccording to claim 3, wherein each of the positioning devices for keypositions comprises an identification panel, a connecting member and afixing member; the fixing member is provided with a notch matching ashape of the guzheng code, the fixing member is installed on an upperend of the guzheng code through the notch during positioning; and theidentification panel is connected to the fixing member through theconnecting member.
 12. The AR-based supplementary teaching system forguzheng according to claim 6, wherein each of the positioning devicesfor key positions comprises an identification panel, a connecting memberand a fixing member; the fixing member is provided with a notch matchinga shape of the guzheng code, the fixing member is installed on an upperend of the guzheng code through the notch during positioning; and theidentification panel is connected to the fixing member through theconnecting member.
 13. The AR-based supplementary teaching system forguzheng according to claim 3, wherein the system further comprises acloud server; the cloud server is in signal connection with a pluralityof data processing devices, and each of the plurality of data processingdevices performs data interaction with other data processing devicesthrough the cloud server.
 14. The AR-based supplementary teaching systemfor guzheng according to claim 6, wherein the system further comprises acloud server; the cloud server is in signal connection with a pluralityof data processing devices, and each of the plurality of data processingdevices performs data interaction with other data processing devicesthrough the cloud server.
 15. The AR-based supplementary teaching methodaccording to claim 9, wherein the system further comprises a sound inputdevice, the sound input device is signally connected to the dataprocessing device; the voice input device is used to collect a sounddata when the user plays and send the sound data to the data processingdevice; the data processing device is configured to identify the sounddata, generate a tone data, retrieve the operation instructionassociated with the tone data, execute the operation instruction, andgenerate the virtual data.
 16. The AR-based supplementary teachingmethod according to claim 17, wherein the data processing devicecomprises a transmission module, an identification module, a storagemodule, and a processing module, wherein the identification module issignally connected to the sound input device, the transmission module,the storage module, and the processing module respectively, theprocessing module is also signally connected to the transmission moduleand the storage module respectively, and the transmission module issignally connected to the AR device; the identification module is usedfor receiving and identifying the sound data to obtain the tone data,and also for receiving and identifying the real scene data to obtain thestring distribution data; the processing module is used for retrievingthe operation instruction associated with the tone data from the storagemodule, executing the operation instruction, and generating the virtualdata according to the operation instruction; and also for controllingthe AR device to superimpose and display the virtual data with the realscene data; the virtual data comprises a string guidance data, afingering guidance data, and a numbered musical notation display data.17. The AR-based supplementary teaching method according to claim 9,wherein the system further comprises a somatosensory device, thesomatosensory device is signally connected to the data processingdevice; the data processing device also used to generate a somatosensoryguidance data according to the operation instruction, and control thesomatosensory device to perform a somatosensory feedback according tothe somatosensory guidance data.
 18. The AR-based supplementary teachingmethod according to claim 17, wherein the somatosensory device comprisesa wristband and four smart fingers; each of the smart fingers isprovided with a pick installation sleeve, a first vibration feedbackdevice and a second vibration feedback device, wherein all firstvibration feedback devices are located on a same side of the pickinstallation sleeves, all second vibration feedback devices are locatedon an opposite side of the pick installation sleeves, and each of thepick installation sleeves is used to install a pick; the wristband isprovided with a somatosensory transmission module, a battery module, anda control module, wherein the battery module supplies power to the firstvibration feedback devices, the second vibration feedback devices, thesomatosensory transmission module, and the control module; the controlmodule is signally connected to the first vibration feedback devices,the second vibration feedback devices, and the somatosensorytransmission module respectively; and the somatosensory transmissionmodule is signally connected to the transmission module; the controlmodule is used for receiving fingering somatosensory guidance data bythe somatosensory transmission module, and controlling the firstvibration feedback devices and the second vibration feedback devices toperform a vibration feedback according to the fingering somatosensoryguidance data.
 19. The AR-based supplementary teaching method accordingto claim 18, wherein a lower surface of each of the pick installationsleeves is provided with an upper mounting member, a first pressuresensor, and a second pressure sensor; the first pressure sensor islocated on a side away from the pick, and the second pressure sensor islocated on a side close to the pick side, the first pressure sensor andthe second pressure sensor are both signally connected to the controlmodule; wherein an upper surface of each of the smart fingers isprovided with a lower mounting member, a first trigger, and a secondtrigger; the first trigger corresponds to the first pressure sensor, andthe second trigger corresponds to the second pressure sensor; the lowermounting member and the upper mounting member are rotatably connected,so that when the pick move the string, the first pressure sensorsqueezes the first trigger or the second pressure sensor squeezes thesecond trigger; wherein the control module is used to receive a pressuredata detected by the first pressure sensor or the second pressuresensor, and send the obtained pressure data to the data processingdevice by the somatosensory transmission module, and the data processingdevice analysis a fingering accuracy according to the pressure data. 20.The AR-based supplementary teaching method according to claim 9, whereineach of the positioning devices for key positions comprises anidentification panel, a connecting member and a fixing member; thefixing member is provided with a notch matching a shape of the guzhengcode, the fixing member is installed on an upper end of the guzheng codethrough the notch during positioning; and the identification panel isconnected to the fixing member through the connecting member.